{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 交通信号灯识别S2TLD--PP-YOLOv2\n",
    "## 1.项目背景\n",
    "本项目交通信号灯识别指的是对行车红、绿、黄灯的识别，识别交通信号灯有以下应用：\n",
    "1. 交通信号灯保障预警\n",
    "2. 闯红灯违章抓拍\n",
    "3. 引导盲人过马路\n",
    "\n",
    "\n",
    "##  2.数据集介绍\n",
    "本项目采用AI Studio提供数据集 \n",
    "\n",
    "[AI Studio S2TLD数据集介绍](https://aistudio.baidu.com/aistudio/datasetdetail/108686)\n",
    "\n",
    "另外CSDN博主「小楞」做了详细信号灯标志牌公开数据集整理：\n",
    "[《信号灯标志牌公开数据集整理及思考整理》](https://blog.csdn.net/qq_33270279/article/details/105127133)\n",
    "\n",
    "以下为其对《Bosch Small Traffic Lights Dataset》整理\n",
    "### 2.1.训练集：\n",
    "\n",
    "•5093图片\n",
    "\n",
    "•大约每2秒钟标注一次\n",
    "\n",
    "•10756带标注的交通灯\n",
    "\n",
    "•红绿灯平局宽度：8.6像素\n",
    "\n",
    "•15种不同的标签（R,R_L,R_R,Y,G,G_L,G_R (遮挡与否*2),off）\n",
    "\n",
    "•170个灯被部分遮挡\n",
    "\n",
    "###  2.2.测试集：\n",
    "\n",
    "•连续8334张图像\n",
    "\n",
    "•大约以15 fps进行注释\n",
    "\n",
    "•13486带标注的交通灯\n",
    "\n",
    "•红绿灯平均宽度：8.5像素\n",
    "\n",
    "•4个标签（红色，黄色，绿色，关闭）\n",
    "\n",
    "•2088灯被部分遮挡\n",
    "\n",
    "### 2.3.场景覆盖情况：\n",
    "\n",
    "•市中心繁忙的街道场景\n",
    "\n",
    "•交通密度变化的郊区多车道\n",
    "\n",
    "•密集的走走停停的交通\n",
    "\n",
    "•道路工程\n",
    "\n",
    "•光照/曝光变化很大\n",
    "\n",
    "•乌云密布的天空，有小雨\n",
    "\n",
    "•忽隐忽现的交通信号灯\n",
    "\n",
    "•多个可见交通灯\n",
    "\n",
    "•可能与交通信号灯混淆的图像部分（例如，大型圆形尾灯）\n",
    "\n",
    "### 2.4.原始图片预览\n",
    "![原始图片预览](https://ai-studio-static-online.cdn.bcebos.com/3a59dc068848427481372002f2f52e85f0facb5e15ea410996f767bdf6c10231)\n",
    "![原始图片预览](https://ai-studio-static-online.cdn.bcebos.com/360f7c913d01441f912fdb3c6964b56a2c436d63a68e4f3caa31e323473aa484)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 3.模型介绍\n",
    "本项目选择PP-YOLOv2进行模型训练，该模型在coco数据集上mAP及预测速度表现领先同类模型。\n",
    "\n",
    "## PP-YOLOv2简介\n",
    "PP-YOLOv2从飞桨明星检测模型PP-YOLO出发，通过增量消融方式逐步添加有助于精度提升且不增加推理耗时的模块达到进一步提升性能的目的。PPYOLOv2（R50）COCO test数据集mAP从45.9%达到了49.5%，相较v1提升了3.6个百分点，FP32 FPS高达68.9FPS，FP16 FPS高达106.5FPS，超越了YOLOv4甚至YOLOv5！而如果使用RestNet101作为骨架网络，PPYOLOv2（R101）的mAP更高达50.3%，并且比同等精度下的YOLOv5x快15.9%！.\n",
    "\n",
    "[10分钟快速上手使用PaddleX2.0——PP-YOLOv2目标检测](https://aistudio.baidu.com/aistudio/projectdetail/3489248?contributionType=1)\n",
    "\n",
    "[目标检测7日打卡营](https://aistudio.baidu.com/aistudio/course/introduce/1617?directly=1&shared=1)\n",
    "：详细介绍PP-YOLO优化历程\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.数据准备与模型训练\n",
    "### 4.1.数据准备\n",
    "本项目采用AI Studio公开数据集[Small Traffic Light Dataset (S2TLD)](https://aistudio.baidu.com/aistudio/datasetdetail/108686)`\n",
    "\n",
    "原始数据压缩包目录如下：\n",
    "\n",
    "![数据目录](https://ai-studio-static-online.cdn.bcebos.com/93b1e79f0a8443238e9e6ce4c37bfc65c5954fffc07843dbb1727f7c08a6cc41)\n",
    "\n",
    "通过如下解压命令或windows本地操作\n",
    "```\n",
    "    1  cd data/ \n",
    "    2  cd data108686/   \n",
    "    3  unzip S2TLD.zip \n",
    "    4  unzip -oq /home/aistudio/data/data108686/S2TLDги1080x1920гй.zip\n",
    "```\n",
    "手动将文件目录重命名为S2TLD_1080_1920\n",
    "将数据文件夹拷贝到work目录"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#拷贝文件并查看文件目录\r\n",
    "!mv data/data108686/S2TLD_1080_1920/ work/\r\n",
    "!tree work/S2TLD_1080_1920 -d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1222 work/S2TLD_1080_1920/train.txt\r\n"
     ]
    }
   ],
   "source": [
    "#去掉文件名中空格，并生成满足PP-YOLOv2要求train.txt \r\n",
    "data_dir = r'work/S2TLD_1080_1920'            #数据所在目录\r\n",
    "file_list = r'work/S2TLD_1080_1920/train.txt'\r\n",
    "\r\n",
    "import os\r\n",
    "with open(file_list,'w') as f:\r\n",
    "    \r\n",
    "    for fn_img in os.listdir(data_dir+'/JPEGImages'):\r\n",
    "        #去掉训练图片文件名中空格    \r\n",
    "        src = data_dir+'/JPEGImages/' + fn_img\r\n",
    "        dst = data_dir+'/JPEGImages/' + fn_img.replace(' ','')\r\n",
    "        if os.path.isfile(src):\r\n",
    "            os.rename(src, dst)\r\n",
    "            \r\n",
    "        img_path = 'JPEGImages/' + fn_img.replace(' ','')\r\n",
    "\r\n",
    "        #去掉训练标签文件名中空格  \r\n",
    "        fn_xml = fn_img.rstrip('.jpg') + '.xml'\r\n",
    "        src = data_dir+'/Annotations/' + fn_xml\r\n",
    "        dst = data_dir+'/Annotations/' + fn_xml.replace(' ','') \r\n",
    "        if os.path.isfile(src):   \r\n",
    "            os.rename(src, dst)\r\n",
    "        annotation_path = 'Annotations/' + fn_xml.replace(' ','') \r\n",
    "            \r\n",
    "        #按voc格式将文件路径写到train.txt文件中    \r\n",
    "        #print(img_path +' '+ annotation_path)\r\n",
    "        f.write(img_path +' '+ annotation_path + '\\n')\r\n",
    "\r\n",
    "#查看文件行数，1222 data/data108686/S2TLD_1080_1920/train.txt  与图片文件个数一致\r\n",
    "!wc -l work/S2TLD_1080_1920/train.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.2.定义数据集\n",
    "请提前安装好paddlex，AI Studio 安装命令如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install paddlex\r\n",
    "!pip install pycocotools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#定义数据\r\n",
    "#import sys\r\n",
    "#sys.path.append('/home/aistudio/external-libraries')\r\n",
    "import paddlex as pdx\r\n",
    "\r\n",
    "#标签类别文件\r\n",
    "label_list = r'work/S2TLD_1080_1920/class.txt'\r\n",
    "\r\n",
    "#定义训练集transforms，采用默认参数\r\n",
    "from paddlex import transforms as T\r\n",
    "train_transforms = T.Compose([\r\n",
    "    T.MixupImage(mixup_epoch=-1), T.RandomDistort(),\r\n",
    "    T.RandomExpand(im_padding_value=[123.675, 116.28, 103.53]), T.RandomCrop(),\r\n",
    "    T.RandomHorizontalFlip(), T.BatchRandomResize(\r\n",
    "        target_sizes=[\r\n",
    "            320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704,\r\n",
    "            736, 768\r\n",
    "        ],\r\n",
    "        interp='RANDOM'), T.Normalize(\r\n",
    "            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\r\n",
    "])\r\n",
    "\r\n",
    "#定义训练集\r\n",
    "train_dataset = pdx.datasets.VOCDetection(data_dir, file_list, label_list,train_transforms)\r\n",
    "\r\n",
    "#定义验证集transforms，采用默认参数\r\n",
    "eval_transforms = T.Compose([\r\n",
    "    T.Resize(\r\n",
    "        target_size=640, interp='CUBIC'), T.Normalize(\r\n",
    "            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\r\n",
    "])\r\n",
    "\r\n",
    "#定义验证集，这里偷懒直接用训练集数据作为验证集\r\n",
    "eval_dataset = pdx.datasets.VOCDetection(data_dir, file_list, label_list,eval_transforms)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.3.模型训练\n",
    "\n",
    "AI Studio提供GPU配置如下,显存32G很棒，查询命令：nvidia-smi\n",
    "\n",
    "![AI Studio GPU配置](https://ai-studio-static-online.cdn.bcebos.com/de5d89ad5d364a6e95d2298cc4840b24415b2bc7880446bd9f90bd4be00150de)\n",
    "\n",
    "train_batch_size=8,显存使用20G\n",
    "\n",
    "![AI Studio GPU配置_训练中](https://ai-studio-static-online.cdn.bcebos.com/c2625cbdf14c4b5eaa718469fed03a0c20930f0fc1184e0285864e05c3791d6d)\n",
    "\n",
    "大于8分钟运行完一个epoch,总计耗时大约80分钟\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#采用PPYOLOv2，backbone选用ResNet50_vd_dcn\r\n",
    "num_classes = len(train_dataset.labels)\r\n",
    "model = pdx.det.PPYOLOv2(num_classes=num_classes, backbone='ResNet50_vd_dcn')\r\n",
    "\r\n",
    "model.train(\r\n",
    "    num_epochs=10,\r\n",
    "    train_dataset=train_dataset,\r\n",
    "    train_batch_size=8,\r\n",
    "    eval_dataset=eval_dataset,\r\n",
    "    pretrain_weights='COCO',\r\n",
    "    learning_rate=0.005 / 12,\r\n",
    "    warmup_steps=1000,\r\n",
    "    warmup_start_lr=0.0,\r\n",
    "    lr_decay_epochs=[105, 135, 150],\r\n",
    "    save_interval_epochs=5,\r\n",
    "    save_dir='work/output/ppyolov2_r50vd_dcn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.4模型评估\n",
    "\n",
    "预测结果\n",
    "\n",
    "![预测结果](https://ai-studio-static-online.cdn.bcebos.com/108f10f83a184464977d213a7996d5ef310d1820d43c4147bf882d12e2e63b6e)\n",
    "\n",
    "真实标注\n",
    "\n",
    "![真是标注](https://ai-studio-static-online.cdn.bcebos.com/f7e1c94769c44cd986e323a33b2b45fbc9d82a7b9f0a48d29ab2c99886464ac0)\n",
    "\n",
    "预测结果\n",
    "\n",
    "![预测结果_夜间](https://ai-studio-static-online.cdn.bcebos.com/0dd964e9ebbb4950b3653ab328c15c079941afcb77f04f7398e52abfc4579c40)\n",
    "\n",
    "真实标注\n",
    "\n",
    "![真是标注_夜间](https://ai-studio-static-online.cdn.bcebos.com/e10b7db5fe6b4f8aafd3af5591e8067bc34c8adde87e4ce48886527f6f739a9d)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#输出一张预测结果\r\n",
    "import paddlex as pdx\r\n",
    "model = pdx.load_model('work/output/ppyolov2_r50vd_dcn/best_model')\r\n",
    "image_name = r'work/S2TLD_1080_1920/JPEGImages/2020-04-0411_09_12.194260936.jpg'\r\n",
    "\r\n",
    "result = model.predict(image_name)\r\n",
    "pdx.det.visualize(image_name, result, threshold=0.5, save_dir='work/output/ppyolov2_r50vd_dcn')\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 5.项目小结\n",
    "（1）本项目基于AI Studio PP-YOLOv2模型，使用coco数据集下的预训练权重，对S2TLD数据集1080X1920分辨率下1222张照片，进行快速建模验证。从训练集预测结果来看，该模型在遮挡情况、夜间预测结果表现较好。\n",
    "\n",
    "（2）本项目只作为快验证，S2TLD数据集下还有720x1280照片4,564张，为保证模型泛化能力应当加入全部数据进行训练，另外需要独立划分验证集与测试集进行模型训练与评估。\n",
    "\n",
    "（3）考虑到交通信号灯标记框占图片比例较小、形状相对固定，可以考虑对anchor进行重新聚类，可能进一步提高模型进度与训练预测效率。[Anchor聚类参考](https://github.com/PaddlePaddle/PaddleX/blob/develop/docs/apis/tools/anchor_clustering.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 6.作者简介\n",
    "作者2013年硕士毕业，硕士期间研究方向为图形学方向。2018年开始接触深度学习。\n",
    "\n",
    "对计算机视觉、3D视觉、OCR识别方向感兴趣，之前做过图片空间分类项目。\n",
    "\n",
    "学习过吴恩达《机器学习》、李飞飞CS231.n课程、[李宏毅课程-机器学习进阶](https://aistudio.baidu.com/aistudio/education/group/info/1979)、[目标检测七日打卡营](https://aistudio.baidu.com/aistudio/education/group/info/1617)、\n",
    "本人[知乎专栏学习笔记](https://www.zhihu.com/people/bright-49-72/posts)\n",
    "\n",
    "联系方式：113278567@qq.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "请点击[此处](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576)查看本环境基本用法.  <br>\n",
    "Please click [here ](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576) for more detailed instructions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
